Benchmarking Natural Language Understanding Services for Building Conversational Agents

We have recently seen the emergence of several publicly available Natural Language Understanding (NLU) toolkits, which map user utterances to structured, but more abstract, Dialogue Act (DA) or Intent specifications, while making this process accessible lay developer. In paper, we present first wide coverage evaluation and comparison some most popular NLU services, on a large, multi-domain (21 domains) dataset 25 K that collected annotated with Entity Type specifications will be released as part submission ( https://github.com/xliuhw/NLU-Evaluation-Data ). The results show classification Watson significantly outperforms other platforms, namely, Dialogflow, LUIS Rasa; though these also perform well. Interestingly, recognition, performs worse due its low Precision (At time producing camera-ready version noticed seemingly recent addition ‘Contextual Entity’ annotation tool Watson, much like e.g. in Rasa. We’d threfore stress paper does not include an feature NLU.). Again, Rasa well task.